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ABSTRACT 

In this paper, we propose a support driven reweighted t\ 
minimization algorithm (SDRL1) that solves a sequence of 
weighted l\ problems and relies on the support estimate accu- 
racy. Our SDRL1 algorithm is related to the IRL1 algorithm 
proposed by Candes, Wakin, and Boyd. We demonstrate that 
it is sufficient to find support estimates with good accuracy 
and apply constant weights instead of using the inverse coef- 
ficient magnitudes to achieve gains similar to those of IRL1. 
We then prove that given a support estimate with sufficient 
accuracy, if the signal decays according to a specific rate, 
the solution to the weighted l\ minimization problem results 
in a support estimate with higher accuracy than the initial 
estimate. We also show that under certain conditions, it is 
possible to achieve higher estimate accuracy when the inter- 
section of support estimates is considered. We demonstrate 
the performance of SDRL1 through numerical simulations 
and compare it with that of IRL1 and standard £\ minimiza- 
tion. 

Index Terms — Compressed sensing, iterative algo- 
rithms, weighted l\ minimization, partial support recovery 

1. INTRODUCTION 

Compressed sensing is a relatively new paradigm for the ac- 
quisition of signals that admit sparse or nearly sparse repre- 
sentations using fewer linear measurements than their ambi- 
ent dimension JT] |2) . 

Consider an arbitrary signal x G and let y E 1" be 
a set of measurements given by y — Ax + e, where A is a 
known n x N measurement matrix, and e denotes additive 
noise that satisfies ||e||2 < e for some known e > 0. Com- 
pressed sensing theory states that it is possible to recover x 
from y (given A) even when n <C N, that is, using very few 
measurements. When x is strictly sparse — i.e., when there 
are only k < n nonzero entries in x — and when e = 0, one 
may recover an estimate x of the signal x by solving the con- 
strained £ minimization problem 



minimize hi o subject to Au 



y- 
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However, £q minimization is a combinatorial problem and 
quickly becomes intractable as the dimensions increase. In- 
stead, the convex relaxation 



minimize ||u||i subject to \\Au — y\U < e 

USER™ 



(BPDN) 



also known as basis pursuit denoise (BPDN) can be used to 
recover an estimate i. Candes, Romberg and Tao 12 and 
Donoho 1 1 1 show that (BPDN) can stably and robustly re- 
cover x from inaccurate and what appears to be "incomplete" 
measurements y = Ax + e if A is an appropriate measure- 
ment matrix, e.g., a Gaussian random matrix such that n > 
k\og(N/k). Contrary to £o minimization, (BPDN) is a con- 
vex program and can be solved efficiently. Consequently, it 
is possible to recover a stable and robust approximation of x 
by solving (BPDN) instead of ([T]) at the cost of increasing the 
number of measurements taken. 

Several works in the literature have proposed alternate al- 
gorithms that attempt to bridge the gap between £q and £\ 
minimization. These include using £ p minimization with < 
p < 1 which has been shown to be stable and robust under 
weaker conditions than those of l\ minimization, see J3] HI 
|5| . Weighted £\ minimization is another alternative if there 
is prior information regarding the support of the signal to-be- 
receovered as it incorporates such information into the recov- 
ery by weighted basis pursuit denoise (w-BPDN) 

minimize ||u||i, w subject to \\Au — y\\2 < e, (w-BPDN) 

u 

where w € [0, 1]^ and ||u||i, w J2i w il u il i s tne weighted 
£\ norm (see J6] UJ 0). Yet another alternative, the itera- 
tive reweighted £\ minimization (IRL1) algorithm proposed 
by Candes, Wakin, and Boyd and studied by Needell ifTOl 
solves a sequence of weighted £\ minimization problems with 



the weights w 



(t) 



V 



where x'f ^ is the solution 



of the (t ~ l)th iteration and w^ 0) = 1 for all i e {1 . . . N}. 

In this paper, we propose a support driven iterative 
reweighted £\ (SDRL1) minimization algorithm that uses 
a small number of support estimates that are updated in every 
iteration and applies a constant weight on each estimate. The 
algorithm, presented in section [2] relies on the accuracy of 
each support estimate as opposed to the coefficient magni- 
tude to improve the signal recovery. While we still lack a 
proof that SDRL1 outperforms £i minimization, we present 
two results in section [3] that motivate SDRL1 and could lead 



towards such a proof. First, we prove that if x belongs to a 
class of signals that satisfy certain decay conditions and given 
a support estimate with accuracy larger than 50%, solving a 
weighted l\ minimization problem with constant weights is 
guaranteed to produce a support estimate with higher accu- 
racy. Second, we show that under strict conditions related 
to the distribution of coefficients in a support estimate, it is 
possible to achieve higher estimate accuracy when the in- 
tersection of support estimates is considered. Finally, we 
demonstrate through numerical experiments in section |4] that 
the performance of our proposed algorithm is similar to that 
oflRLl. 

2. ITERATIVE REWEIGHTED l x MINIMIZATION 

In this section, we give an overview of the IRL1 algorithm, 
proposed by Candes, Wakin, and Boyd [9] and present our 
proposed support driven reweighted l\ (SDRL1) algorithm. 
2.1. The IRL1 algorithm 

IRL1 algorithm solves a sequence of (w-BPDN) problems 
where the weights are chosen according to = rgfrp^ • Here 
Xi is an estimate of the signal coefficient at index i (from the 
previous iteration) and a is a stability parameter. The choice 
of a affects the stability of the algorithm and different vari- 
ations are proposed for the sparse, compressible, and noisy 
recovery cases. The algorithm is summarized Algorithm[T] 

Algorithm 1 IRL1 algorithm |9| 
1 : Input y = Ax + e 

2: Output X® 

3: Initialize w| 0) = 1 for all % G {1 . . .N}, a 

t = 0, as(°) = 
4: while \\xW - x^-Vy < Toljlart*- 1 ^ || 2 do 

5: t = t+l 

6: a;w = argmin w s -t- \\Au — y\\ 2 < e 
7: w i = I- fl 

b \Xi\+a 

8: end while 



The rationale behind choosing the weights inversely pro- 
portional to the estimated coefficient magnitude comes from 
the fact that large weights encourage small coefficients and 
small weights encourage large coefficients. Therefore, if the 
true signal were known exactly, then the weights would be set 
equal to w* = r^-r . Otherwise, weighting according to an ap- 
proximation of the true signal and iterating was demonstrated 
to result in better recovery capabilities than standard i\ min- 
imization. In ifTUl , the error bounds for IRL1 were shown 
to be tighter than those of standard l\ minimization. How- 
ever, aside from empirical studies, no provable results have 
yet been obtained to show that IRL1 outperforms standard t\, 

2.2. Support driven reweighted l\ (SDRL1) algorithm 

In 10, we showed that solving the weighted l\ problem with 
constant weights applied to a support estimate set T has bet- 
ter recovery guarantees than standard l\ minimization when 



the T is at least 50% accurate. Moreover, we showed in ifTTI 
that using multiple weighting sets improves on our previous 
result when additional information on the support estimate 
accuracy is available. Motivated by these works, we pro- 
pose the SDRL1 algorithm, a support accuracy driven iter- 
ative reweighted l\ minimization algorithm, which identifies 
two support estimates that are updated in every iteration and 
applies constant weights on these estimates. The SDRL1 al- 
gorithm relies on the support estimate accuracy as opposed 
to the coefficient magnitude. The algorithm is presented in 
Algorithm |2] 

Algorithm 2 Support driven reweighted i\ (SDRL1) algo- 
rithm^ 

1 : Input y = Ax + e 

2: Output X^> 

3: Initialize^ = 0.99, k = nlog(N/n)/2, 

wi = 0.5, lu 2 = 0, Tol, T x = 0, ft = 0, 
t = 0, s<W = 0, = 

4: while ||x (t) -a: (t - 1) ||2 < TolH^'" 1 ) || 2 do 

5: t = t+l 

6: ft = supp^'- 1 } | s ( t -i))nTi 
7: Set the weights equal to 

r i, ie Tfnfr 

Wi — < Wi, i G Ti n fl c 

8: a^ 4 ' = arg min ||u|| i w s.t. IIAtt — ylU < e 
9: I = min|A| s.t. ||x^|| 2 > P\\x {t) \\ 2 , 

s(*) = min{Z, k} 
10: Ti = supp(a;( t )| s ( t) ) 
li: end while 



Note that we use two empirical parameters to control the 
size of the support estimate T%, The first parameter k approx- 
imates the minimum sparsity level recoverable by (BPDN). 
The second parameter I is the number of largest coefficients 
of x^ that contribute an ad hoc percentage p of the signal 
energy. The size of T\ is set equal to the minimum of k and I. 

3. MOTIVATING THEORETICAL RESULTS 

The SDRL1 algorithm relies on two main premises. The first 
is the ability to improve signal recovery using a sufficiently 
accurate support estimate by solving a weighted l\ minimiza- 
tion problem with constants weights. The second is the in- 
tersection set of two support estimates has at least the higher 
accuracy of either set. 

Let x G R be an arbitrary signal and suppose we collect 
n -C N linear measurements y = Ax, A € M nxAr where n 
is small enough (or k is large enough) that it is not possible 
to recover x exactly by solving (BPDN) with e = 0. Denote 
by x the solution to (BPDN), and by x u the solution to (w- 



BPDN) with weight cu applied to a support estimate set T. 
Let Xk be the best fc-term approximation of x and denote by 
To = supp(xfc) the support set of Xk- 

Proposition 3.1. Suppose that T is of size k with accuracy 
(with respect to To) ao = for some integer k/2 < sq < k. 

_ 2 

If A has RIP with constant <5( a +i)fc < ° + ^ 2 for some a > 1 
and 7 = uj + (1 — lu)^/2 — 2a , and if there exists a positive 
integer d\ such that 

\x(s Q + di)\ > (b}r) + l)\\x T c\\ 1 + (l-u))ri\\x T < sn f e \\i, (2) 

where rj — r) u (ao) is a well behaved constant, then the set 
S = supp(a; So +d 1 ) is contained in T u — supp(x^). 

Remark 3.1.1. The constant rj u (a) is given explicitly by 

, . 2 (VI + 8ak + \faJ\ - 5( a +l)fe ) 

r]w(<x) = — . 

V«Vl - <5(a+i)fc - (u + (1 - w)V2 - 2a) Vl + 5ak 



Proposition 3.1 shows that if the signal x satisfies condi- 



P roof outline. The proof of Proposition 3.1 is a direct exten- 
sion of our proof of Proposition 3.2 in Mill . In particular, we 
want to find the conditions on the signal x and the matrix A 
which guarantee that the set S = supp(x Sn+( j 1 ) is a subset of 
T u = supp(i^). This is achieved when x u satisfies 



minlx^O')! > max|x w (?')|. 



(3) 



Since A has RIP with 5( a +i)k < a+j 2 ' ^ nas Space 
property (NSP) |12| of order k, i.e., for any h G Af(A), Ah = 
0, then \\h\\i < co||/it„ c ||i, with cq = 1 - 



Define h = x u — x, then /i G M{A) and one can show that 

INi < V {u\W § ||i + (l-w)||a; T . n5 ; e ||i) (4) 

In other words, (w-BPDN) is £i-^i instance optimal with 
these error bounds. The proof of this fact is a direct extension 
of the tx-li instance optimality of (BPDN) as shown in lfl"2"l 
and we omit the details here. Next, we rewrite Q as 

\\h Ta \\i < {ur) + l)\\x T§ \\i + {l-u)r)\\x TSnf A\x-\\x% s \\x. 

To complete the proof, we make the following observations: 

(i) Taxa.\x u '{j)\ > min|x(j')| — iaax\x(J) — SP{j)\, 
jes jes jeS 

(ii) \\x%c\\i > max j6T c \x u (j)\ 

which after some manipulations -see JTl |, proof of Prop. 3.2 
for details of a similar calculation in a different setting- imply 

min |£ w (j)| > maxj 6 T c +rnin|x(j)| 

jes " jes 

-(wr)+ 1)||xt c||i + (1 - u))r]\\x TSn f c \\i. 

(5) 

Finally, we observe from |5]l that ([3]) holds, i.e., S C T u , if 



\x(s +di)\ > (urj + l)||a; T c||i + (1 - u))r)\\x T a n q, a \\ 



tion (|2) and ^ > 0.5, then the support of the largest k coeffi- 
cients of x u contains at least the support of the largest sq + d\ 
coefficients of x for some positive integer d\. 

Next we present a proposition where we focus on an ide- 
alized scenario: Suppose that the events JSj := {i G T}, for 
i G {1, . . . , N} and T C {1, . . . , N}, are independent and 
have equal probability with respect to an appropriate discrete 
probability measure P. In this case, we show below, that the 
accuracy of £1 = T n T u is at least as high as the higher of the 
accuracies of T and T u . For simplicity, we use the notation 
P(T |f ) to denote P(i G T \i G T ). 

Proposition 3.2. Let x be an arbitrary signal in R and de- 
note by To the support of the best k-term approximation of x 
Let the sets T and T w be each of size k and suppose that T 
and T u contain the support of the largest sq and s\ > sq coef- 
ficients of x, respectively. Define the set Q — TnT u . Given a 
discrete probabilty measure P, the events Ei := {i G T}, for 
i G {1, . . . , N} and T C {1, . . . , N}, are independent and 
equiprobable. Then, for p :— P(T U \T) > ^t, the accuracy of 
the set n is given by 

P(T |Q) = if. 

Proof outline. The proof follows directly using elementary 
tools in probability theory. In particular, we have P(Tq|T) = 
f, and P(T |TL) = f. Define p = P(T U \T) > f, it is 
easy to see that P(T n T|T n T u ) = ^ which leads to 

p(t n o) = P(r n r u )P(r n t\t n t l 

Consequently, P(T |fi) - P(T ° n0) - 



P(T^|T)Pr(T) 



Proposition 



Pr(T |O) 



3.2 



1. T 



, Sj^ so so 

N si ~ N- 
so/N 1 so 

p(k/N) ~ p k ■ 

□ 



indicates that as Pr(TjT) -> ^, then 
'herefore, when x satisfies |2| it could be 
beneficial to solve a weighted l\ problem where we can take 
advantage of the possible improvement in accuracy on the set 
TnT u . Finally, we note that there are more complex depen- 
dencies between the entries of T and T u of Algorithm u\ for 



□ 



which Proposition 3.2 does not account. 

4. NUMERICAL RESULTS 

We tested our SDRL1 algorithm by comparing its perfor- 
mance with IRL1 and standard £j minimization in recovering 
synthetic signals x of dimension N — 2000. We first recover 
sparse signals from compressed measurements of x using ma- 
trices A with i.i.d. Gaussian random entries and dimensions 
fix JV where n G {iV/10, N/4, N/2}. The sparsity of the 
signal is varied such that k/n G {0.1, 0.2, 0.3, 0.4, 0.5}. To 
quantify the reconstruction performance, we plot in Figure 
[T]the percentage of successful recovery averaged over 100 
realizations of the same experimental conditions. The figure 
shows that both the proposed algorithm and IRL1 have a 
comparable performance which is far better than standard ii 
minimization. 



L minimization 




Fig. 1. Comparison of the percentage of exact recovery of sparse signals between the proposed SDRL1 algorithm, IRL1 |9|, 
and standard l\ minimization. The signals have an ambient dimension N = 2000 and the sparsity and number of measurements 
are varied. The results are averaged over 100 experiments. 
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Fig. 2. Histogram of the ratio of the mean squared error (MSE) between the proposed SDRL1 and IRL1 |9| for the recovery of 
compressible signals. The signals x follow a power law decay such that \xi\ — ci~ p , for constant c and exponent p. 



Next, we generate compressible signals with power law 
decay such that x(i) = ci~ v for some constant c and de- 
cay power p. We consider the case where n/N = 0.1 and 
the decay power p E {1.1, 1.5, 2} and plot the ratio of the 
reconstruction error of SDRL1 over that of IRL1. Figure [2] 
shows the histograms of the ratio for 100 experiments each. 
Note that a ratio smaller than one means that our algorithm 
has a smaller reconstruction error than that of IRL1. The his- 
tograms indicate that both algorithms have a comparable per- 
formance for signals with different decay rates. 
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